We propose Universal Document Processing (UDOP), a foundation Document AI model which unifies text, image, and layout modalities together with varied task formats, including document understanding and generation. UDOP leverages the spatial correlation between textual content and document image to model image, text, and layout modalities with one uniform representation. With a novel Vision-Text-Layout Transformer, UDOP unifies pretraining and multi-domain downstream tasks into a prompt-based sequence generation scheme. UDOP is pretrained on both large-scale unlabeled document corpora using innovative self-supervised objectives and diverse labeled data. UDOP also learns to generate document images from text and layout modalities via masked image reconstruction. To the best of our knowledge, this is the first time in the field of document AI that one model simultaneously achieves high-quality neural document editing and content customization. Our method sets the state-of-the-art on 9 Document AI tasks, e.g., document understanding and QA, across diverse data domains like finance reports, academic papers, and websites. UDOP ranks first on the leaderboard of the Document Understanding Benchmark (DUE).

本文提出了一个紧凑的系统OpenPneu，以支持软机器人多腔的气动驱动。系统中使用微型泵来生成气流，因此不需要额外的输入，因为需要压缩空气。我们的系统执行模块化设计以提供良好的可扩展性，这已在具有十个空气通道的原型上证明。OpenPNEU的每个空气通道都配备了通货膨胀和通气功能，可提供从正到负的全范围压力供应，最大流速为1.7 L/min。我们的系统内置了对压力的高精度闭环控制，以实现稳定而有效的动态性能。提供了Python中的开源控制接口和API。我们还证明了OpenPneu在三个软机器人系统上的功能，最多10个腔室。

尽管在文档理解方面取得了成功，但由于计算中的几个挑战以及如何有效吸收长期多模式输入，因此长期文档理解的实际任务在很大程度上尚未探索。大多数基于变压器的方法仅处理简短的文档，并且由于其过度的计算和内存限制，因此仅使用文本信息来引起注意。为了解决长期文档理解中的这些问题，我们探索了处理1D和新的2D位置引人入胜的不同方法，并以本质上的背景缩短了。实验结果表明，我们提出的模型基于各种评估指标具有此任务的优势。此外，我们的模型仅对注意力进行更改，因此很容易适应任何基于变压器的体系结构。

软机器人由于其机械合规性可以安全地与环境相互作用。在现代的软机器人的现代设计中，自我碰撞也用于在不同的任务中提高其性能。但是，开发一个可以很好地处理碰撞响应的高效且可靠的模拟器，仍然是软机器人技术研究中的一项艰巨任务。本文基于几何优化提供了一个碰撞感知的模拟器，其中我们开发了一种高效且逼真的碰撞检查 /响应模型，该模型包含了超弹性材料特性。软机器人的驱动变形和碰撞响应都是基于几何目标的。可以通过最小化基于几何的目标函数来获得软机器人的无碰撞主体。与基于FEA的物理模拟不同，所提出的管道的计算成本要低得多。此外，在处理具有较大体积变化的软机器人时，适用自适应重新捕获以提高收敛性。在不同的软机器人上进行了实验测试，以验证我们的方法的性能。

由于其有效的模型架构以及大规模未标记的扫描/数字出生的文件的优势，在各种视觉上丰富的文档理解任务中已经证明了文本和布局的预先培训。我们提出了具有新的预培训任务的Layoutlmv2架构，以在单个多模态框架中模拟文本，布局和图像之间的交互。具体地，对于双流多模态变压器编码器，LayOutLMV2不仅使用现有屏蔽的视觉语言建模任务，还使用新的文本图像对齐和文本图像匹配任务，这使得它更好地捕获跨模块交互在预训练阶段。同时，它还将空间感知的自我注意机制集成到变压器架构中，以便模型可以完全理解不同文本块之间的相对位置关系。实验结果表明，LayoutLMV2优于大幅度的LayOutlm，并在大量下游的下游富有的文件理解任务中实现了新的最先进的结果，包括Funsd（0.7895 $ \至0.8420美元），电源线（0.9493 $ \至0.9601美元），Srie（0.9524 $ \至0.9781美元），Kleister-NDA（0.8340 $ \ 0.8520美元），RVL-CDIP（0.9443 $ \至0.9564美元），DOCVQA（0.7295 $ \至0.8672美元） 。我们使我们的模型和代码公开可用于\ url {https://aka.ms/layoutlmv2}。

In this paper, we propose a robust 3D detector, named Cross Modal Transformer (CMT), for end-to-end 3D multi-modal detection. Without explicit view transformation, CMT takes the image and point clouds tokens as inputs and directly outputs accurate 3D bounding boxes. The spatial alignment of multi-modal tokens is performed implicitly, by encoding the 3D points into multi-modal features. The core design of CMT is quite simple while its performance is impressive. CMT obtains 73.0% NDS on nuScenes benchmark. Moreover, CMT has a strong robustness even if the LiDAR is missing. Code will be released at https://github.com/junjie18/CMT.

Knowledge graphs (KG) have served as the key component of various natural language processing applications. Commonsense knowledge graphs (CKG) are a special type of KG, where entities and relations are composed of free-form text. However, previous works in KG completion and CKG completion suffer from long-tail relations and newly-added relations which do not have many know triples for training. In light of this, few-shot KG completion (FKGC), which requires the strengths of graph representation learning and few-shot learning, has been proposed to challenge the problem of limited annotated data. In this paper, we comprehensively survey previous attempts on such tasks in the form of a series of methods and applications. Specifically, we first introduce FKGC challenges, commonly used KGs, and CKGs. Then we systematically categorize and summarize existing works in terms of the type of KGs and the methods. Finally, we present applications of FKGC models on prediction tasks in different areas and share our thoughts on future research directions of FKGC.

Few Shot Instance Segmentation (FSIS) requires models to detect and segment novel classes with limited several support examples. In this work, we explore a simple yet unified solution for FSIS as well as its incremental variants, and introduce a new framework named Reference Twice (RefT) to fully explore the relationship between support/query features based on a Transformer-like framework. Our key insights are two folds: Firstly, with the aid of support masks, we can generate dynamic class centers more appropriately to re-weight query features. Secondly, we find that support object queries have already encoded key factors after base training. In this way, the query features can be enhanced twice from two aspects, i.e., feature-level and instance-level. In particular, we firstly design a mask-based dynamic weighting module to enhance support features and then propose to link object queries for better calibration via cross-attention. After the above steps, the novel classes can be improved significantly over our strong baseline. Additionally, our new framework can be easily extended to incremental FSIS with minor modification. When benchmarking results on the COCO dataset for FSIS, gFSIS, and iFSIS settings, our method achieves a competitive performance compared to existing approaches across different shots, e.g., we boost nAP by noticeable +8.2/+9.4 over the current state-of-the-art FSIS method for 10/30-shot. We further demonstrate the superiority of our approach on Few Shot Object Detection. Code and model will be available.

Graph Neural Networks (GNNs) have shown satisfying performance on various graph learning tasks. To achieve better fitting capability, most GNNs are with a large number of parameters, which makes these GNNs computationally expensive. Therefore, it is difficult to deploy them onto edge devices with scarce computational resources, e.g., mobile phones and wearable smart devices. Knowledge Distillation (KD) is a common solution to compress GNNs, where a light-weighted model (i.e., the student model) is encouraged to mimic the behavior of a computationally expensive GNN (i.e., the teacher GNN model). Nevertheless, most existing GNN-based KD methods lack fairness consideration. As a consequence, the student model usually inherits and even exaggerates the bias from the teacher GNN. To handle such a problem, we take initial steps towards fair knowledge distillation for GNNs. Specifically, we first formulate a novel problem of fair knowledge distillation for GNN-based teacher-student frameworks. Then we propose a principled framework named RELIANT to mitigate the bias exhibited by the student model. Notably, the design of RELIANT is decoupled from any specific teacher and student model structures, and thus can be easily adapted to various GNN-based KD frameworks. We perform extensive experiments on multiple real-world datasets, which corroborates that RELIANT achieves less biased GNN knowledge distillation while maintaining high prediction utility.

This paper focuses on designing efficient models with low parameters and FLOPs for dense predictions. Even though CNN-based lightweight methods have achieved stunning results after years of research, trading-off model accuracy and constrained resources still need further improvements. This work rethinks the essential unity of efficient Inverted Residual Block in MobileNetv2 and effective Transformer in ViT, inductively abstracting a general concept of Meta-Mobile Block, and we argue that the specific instantiation is very important to model performance though sharing the same framework. Motivated by this phenomenon, we deduce a simple yet efficient modern \textbf{I}nverted \textbf{R}esidual \textbf{M}obile \textbf{B}lock (iRMB) for mobile applications, which absorbs CNN-like efficiency to model short-distance dependency and Transformer-like dynamic modeling capability to learn long-distance interactions. Furthermore, we design a ResNet-like 4-phase \textbf{E}fficient \textbf{MO}del (EMO) based only on a series of iRMBs for dense applications. Massive experiments on ImageNet-1K, COCO2017, and ADE20K benchmarks demonstrate the superiority of our EMO over state-of-the-art methods, \eg, our EMO-1M/2M/5M achieve 71.5, 75.1, and 78.4 Top-1 that surpass \textbf{SoTA} CNN-/Transformer-based models, while trading-off the model accuracy and efficiency well.